Autofocusing camera and systems

ABSTRACT

Apparatuses and methods for focusing a camera are disclosed. For example, an apparatus may be coupled to a camera for focusing the camera. The apparatus includes a vision sensor coupled to a processor and configured to capture a view. The processor configured to receive a selection of an area of interest in the view. The apparatus further includes a distance measurement unit coupled to the processor and configured to measure a distance to the area of interest for adjusting the camera&#39;s focus.

TECHNICAL FIELD

The present disclosure relates generally to cameras and systems withfast autofocusing and automatic focus tracking capabilities.

BACKGROUND

Cameras find use in many environments including, for example,surveillance systems, sports, unmanned aerial vehicles (UAV), etc.Whether to take a sharp image or an intentionally blurry image, one mustadjust the camera to the correct focus. Most cameras today include anumber of lenses or lens groups that can move with respect to oneanother, thereby providing for automatic focusing.

Several common autofocusing techniques exist. For example, a camera mayinclude a device for measuring the distance to an object andautomatically focusing on the object based on the measured distance. Thedistance measurement device may include an infrared light or laseremitter and a light sensor that senses the infrared light or laser. Thetime of flight (TOF), i.e., from the time the light is emitted from theemitter to the time the sensor senses the light, reflects the distancebetween the device and the object. Some distance measurement devices mayalso utilize ultrasound instead of light. With the measured distance, acontroller (such as a computer) in the camera can send signals to motorsthat drive and move the lenses to achieve focus.

Some cameras employ a phase detection method to adjust focus. A mirrorreflects the image of the object onto two phase sensors, and a computercompares the two reflected images sensed by the sensors. Focus occurswhen the two reflected images are identical.

Another way of autofocusing, known as contrast detection, involvesdetection of contrast and finding the position of the lenses thatprovides the best contrast. As the lenses or lens groups move, therebychanging focus, the camera takes images of an object, and the computerassociated with the camera analyzes the images and compares contrastsbetween consecutive images. Increase in the contrast between consecutiveimages suggests the lenses are moving in the correct direction forimproving focus, and the position of the lenses that generates the imagewith the highest contrast provides the best focus.

Each method has advantages and disadvantages. Contrast detectionrequires analysis of many images as the lenses move back and forth andis therefore slow. Distance measurement and phase detection methods bothtake much less time. But the distance measurement method can onlydetermine the distance from the camera to the closest object in the viewand fails when one wants to take a picture with a focus on an objectfurther in the view. The phase detection method can achieve focus withprecision rather quickly, but requires complex and expensiveconstruction of the camera, because the camera must include multipleautofocus sensors which each include its own lens and photodetector. Inaddition, the number of autofocus sensors limits the number of areas tofocus on in the view. Two autofocus sensors, for example, means thecamera can only focus on one part of the image. Raising the number offocus points would further raise the price of the camera.

Many cameras combine these autofocusing methods. A typical combinationincludes the distance measurement method or phase detection method as afirst step to quickly get the camera in the ballpark of focus, followedwith contrast detection to fine tune the focus.

These autofocusing methods work well when taking static pictures, butnot so well in moving environments, where objects at different distancesmove with time. Especially when shooting a video, the camera must adjustand track its focus in real-time with objects moving. Manual focusingremains necessary in such situations.

Accordingly, there exists a need for fast, precise, and inexpensiveautofocusing and focus tracking technology adapted for variousenvironments.

SUMMARY

Consistent with embodiments of the present disclosure, an apparatus isprovided for focusing a camera. The apparatus includes a vision sensorcoupled to a processor and configured to capture a view. The processorconfigured to receive a selection of an area of interest in the view.The apparatus further includes a distance measurement unit coupled tothe processor and configured to measure a distance to the area ofinterest for adjusting the camera's focus.

There is also provided a method for focusing a camera. First, a visionsensor captures a view. Then, an area of interest in the captured viewis selected. A distance measurement unit measures the distance of thearea of interest from itself. And the camera's focus is adjusted basedon the measured distance.

There is further provided a movable object that includes a camera, anauxiliary focusing device including a vision sensor and a distancemeasurement unit, and a processor. The processor is configured to causethe vision sensor to capture a view, receive a selection of an area ofinterest, cause the distance measurement unit to measure a distance tothe area of interest, and cause adjustment of the camera's focus basedon the measured distance.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an exemplary movable object configured in accordance withembodiments of the present disclosure.

FIG. 2A shows an exemplary terminal configured in accordance withembodiments of the present disclosure.

FIG. 2B shows another exemplary terminal configured in accordance withembodiments of the present disclosure.

FIG. 3 shows an exemplary system-on-chip controller configured inaccordance with embodiments of the present disclosure.

FIG. 4 shows an exemplary auxiliary focusing device configured inaccordance with embodiments of the present disclosure.

FIG. 5 shows a flow diagram of an exemplary autofocusing process inaccordance with embodiments of the present disclosure.

FIGS. 6A and 6B illustrates an exemplary focusing technique consistentwith embodiments of the present disclosure.

FIG. 7 shows a flow diagram of another exemplary autofocusing process inaccordance with embodiments of the present disclosure.

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers refer to the same orsimilar parts. While several illustrative embodiments are describedherein, modifications, adaptations and other implementations arepossible. For example, substitutions, additions or modifications may bemade to the components illustrated in the drawings. Accordingly, thefollowing detailed description is not limited to the disclosedembodiments and examples. Instead, the proper scope is defined by theappended claims.

Consistent with embodiments of the present disclosure, there areprovided cameras or camera systems that can quickly and automaticallyfocus on one or more objects or areas for taking a picture or shooting avideo. In addition to taking static images, these cameras may be used insports, surveillance systems, Unmanned Aerial Vehicles (UAVs), etc.,where one or more of the objects and the cameras may move. The camerasmay be mounted on other devices, such as cars, UAVs, bikes, helmets,etc., or on a person, or mounted on a handheld device to be held inhand.

As an example, FIG. 1 shows a movable object 10 that may be configuredto move or travel within an environment. Movable object 10 may be a UAVor any other suitable object, device, mechanism, system, or machineconfigured to travel on or within a suitable medium (e.g., a surface,air, water, rails, space, underground, etc.). Movable object 10 may alsobe other types of movable object (e.g., wheeled objects, nauticalobjects, locomotive objects, other aerial objects, etc.). As usedherein, the term UAV refers to an aerial device configured to beoperated and/or controlled automatically (e.g., via an electroniccontrol system) and/or manually by off-board personnel.

Movable object 10 includes one or more propulsion devices 12 and may beconfigured to carry a payload 14. Payload 14 may be connected orattached to movable object 10 by a carrier 16, which may allow for oneor more degrees of relative movement between payload 14 and movableobject 10. Payload 14 may also be mounted directly to movable object 10without carrier 16. Movable object 10 also includes a sensing system 18,a communication system 20, and a controller 22 in communication with theother components.

Movable object 10 may include one or more (e.g., 1, 2, 3, 3, 4, 5, 10,15, 20, etc.) propulsion devices 12 positioned at various locations (forexample, top, sides, front, rear, and/or bottom of movable object 10)for propelling and steering movable object 10. Propulsion devices 12 aredevices or systems operable to generate forces for sustaining controlledflight. Propulsion devices 12 may share or may each separately includeor be operatively connected to a power source, such as a motor (e.g., anelectric motor, hydraulic motor, pneumatic motor, etc.), an engine(e.g., an internal combustion engine, a turbine engine, etc.), a batterybank, etc., or a combination thereof. Each propulsion device 12 may alsoinclude one or more rotary components 24 drivably connected to the powersource and configured to participate in the generation of forces forsustaining controlled flight. For instance, rotary components 24 mayinclude rotors, propellers, blades, nozzles, etc., which may be drivenon or by a shaft, axle, wheel, hydraulic system, pneumatic system, orother component or system configured to transfer power from the powersource. Propulsion devices 12 and/or rotary components 24 may beadjustable (e.g., tiltable) with respect to each other and/or withrespect to movable object 10. Alternatively, propulsion devices 12 androtary components 24 may have a fixed orientation with respect to eachother and/or movable object 10. In some embodiments, each propulsiondevice 12 may be of the same type. In other embodiments, propulsiondevices 12 may be of multiple different types. In some embodiments, allpropulsion devices 12 may be controlled in concert (e.g., all at thesame speed and/or angle). In other embodiments, one or more propulsiondevices may be independently controlled with respect to, e.g., speedand/or angle.

Propulsion devices 12 may be configured to propel movable object 10 inone or more vertical and horizontal directions and to allow movableobject 10 to rotate about one or more axes. That is, propulsion devices12 may be configured to provide lift and/or thrust for creating andmaintaining translational and rotational movements of movable object 10.For instance, propulsion devices 12 may be configured to enable movableobject 10 to achieve and maintain desired altitudes, provide thrust formovement in all directions, and provide for steering of movable object10. In some embodiments, propulsion devices 12 may enable movable object10 to perform vertical takeoffs and landings (i.e., takeoff and landingwithout horizontal thrust). In other embodiments, movable object 10 mayrequire constant minimum horizontal thrust to achieve and sustainflight. Propulsion devices 12 may be configured to enable movement ofmovable object 10 along and/or about multiple axes.

Payload 14 includes a sensory device 19. Sensory device 19 may includedevices for collecting or generating data or information, such assurveying, tracking, and capturing images or video of targets (e.g.,objects, landscapes, subjects of photo or video shoots, etc.). Sensorydevice 19 may include imaging devices configured to gather data that maybe used to generate images. The imaging devices may include photographiccameras, video cameras, infrared imaging devices, ultraviolet imagingdevices, x-ray devices, ultrasonic imaging devices, radar devices, etc.Sensory device 19 may also or alternatively include devices forcapturing audio data, such as microphones or ultrasound detectors.Sensory device 19 may also or alternatively include other suitablesensors for capturing visual, audio, and/or electromagnetic signals.

Carrier 16 may include one or more devices configured to hold payload 14and/or allow payload 14 to be adjusted (e.g., rotated) with respect tomovable object 10. For example, carrier 16 may be a gimbal. Carrier 16may be configured to allow payload 14 to be rotated about one or moreaxes, as described below. In some embodiments, carrier 16 may beconfigured to allow payload 14 to rotate about each axis by 360° toallow for greater control of the perspective of payload 14. In otherembodiments, carrier 16 may limit the range of rotation of payload 14 toless than 360° (e.g., ≤270°, ≤210°, ≤180, ≤120°, ≤90°, ≤45°, ≤30°, ≤15°,etc.) about one or more of its axes.

Carrier 16 may include a frame assembly 26, one or more actuator members28, and one or more carrier sensors 30. Frame assembly 26 may beconfigured to couple payload 14 to movable object 10 and, in someembodiments, to allow payload 14 to move with respect to movable object10. In some embodiments, frame assembly 26 may include one or moresub-frames or components movable with respect to each other. Actuatormembers 28 are configured to drive components of frame assembly 26relative to each other to provide translational and/or rotational motionof payload 14 with respect to movable object 10. In other embodiments,actuator members 28 may be configured to directly act on payload 14 tocause motion of payload 14 with respect to frame assembly 26 and movableobject 10. Actuator members 28 may be or may include suitable actuatorsand/or force transmission components. For example, actuator members 28may include electric motors configured to provide linear and/orrotational motion to components of frame assembly 26 and/or payload 14in conjunction with axles, shafts, rails, belts, chains, gears, and/orother components.

Carrier sensors 30 may include devices configured to measure, sense,detect, or determine state information of carrier 16 and/or payload 14.State information may include positional information (e.g., relativelocation, orientation, attitude, linear displacement, angulardisplacement, etc.), velocity information (e.g., linear velocity,angular velocity, etc.), acceleration information (e.g., linearacceleration, angular acceleration, etc.), and or other informationrelating to movement control of carrier 16 or payload 14, eitherindependently or with respect to movable object 10. Carrier sensors 30may include one or more types of suitable sensors, such aspotentiometers, optical sensors, visions sensors, magnetic sensors,motion or rotation sensors (e.g., gyroscopes, accelerometers, inertialsensors, etc.). Carrier sensors 30 may be associated with or attached tovarious components of carrier 16, such as components of frame assembly26 or actuator members 28, or to movable object 10. Carrier sensors 30may be configured to communicate data and information with controller 22via a wired or wireless connection (e.g., RFID, Bluetooth, Wi-Fi, radio,cellular, etc.). Data and information generated by carrier sensors 30and communicated to controller 22 may be used by controller 22 forfurther processing, such as for determining state information of movableobject 10 and/or targets.

Carrier 16 may be coupled to movable object 10 via one or more dampingelements configured to reduce or eliminate undesired shock or otherforce transmissions to payload 14 from movable object 10. Dampingelements may be active, passive, or hybrid (i.e., having active andpassive characteristics). Damping elements may be formed of any suitablematerial or combinations of materials, including solids, liquids, andgases. Compressible or deformable materials, such as rubber, springs,gels, foams, and/or other materials may be used as damping elements. Thedamping elements may function to isolate payload 14 from movable object10 and/or dissipate force propagations from movable object 10 to payload14. Damping elements may also include mechanisms or devices configuredto provide damping effects, such as pistons, springs, hydraulics,pneumatics, dashpots, shock absorbers, and/or other devices orcombinations thereof.

Sensing system 18 may include one or more sensors associated with one ormore components or other systems of movable device 10. For instance,sensing system 18 may include sensors for determining positionalinformation, velocity information, and acceleration information relatingto movable object 10 and/or targets. In some embodiments, sensing system18 may also include carrier sensors 30. Components of sensing system 18may be configured to generate data and information that may be used(e.g., processed by controller 22 or another device) to determineadditional information about movable object 10, its components, and/orits targets. Sensing system 18 may include one or more sensors forsensing one or more aspects of movement of movable object 10. Forexample, sensing system 18 may include sensory devices associated withpayload 14 as discussed above and/or additional sensory devices, such asa positioning sensor for a positioning system (e.g., GPS, GLONASS,Galileo, Beidou, GAGAN, RTK, etc.), motion sensors, inertial sensors(e.g., IMU sensors, MIMU sensors, etc.), proximity sensors, imagesensors, etc. Sensing system 18 may also include sensors configured toprovide data or information relating to the surrounding environment,such as weather information (e.g., temperature, pressure, humidity,etc.), lighting conditions (e.g., light-source frequencies), airconstituents, or nearby obstacles (e.g., objects, structures, people,other vehicles, etc.).

Communication system 20 may be configured to enable communication ofdata, information, commands, and/or other types of signals betweencontroller 22 and off-board entities, such as terminal 32, a smartphone,or another suitable entity. Communication system 20 may include one ormore components configured to send and/or receive signals, such asreceivers, transmitter, or transceivers, that are configured for one-wayor two-way communication. Components of communication system 20 may beconfigured to communicate with off-board entities via one or morecommunication networks, such as radio, cellular, Bluetooth, Wi-Fi, RFID,and/or other types of communication networks usable to transmit signalsindicative of data, information, commands, and/or other signals. Forexample, communication system 20 may be configured to enablecommunication between devices for providing input for controllingmovable object 10 during flight, such as a control terminal (“terminal”)32.

Controller 22 may be configured to communicate with various devicesonboard movable object 10, such as communication system 20 and sensingsystem 18. Controller 22 may also communicate with a positioning system(e.g., a global navigation satellite system, or GNSS) (not pictured) toreceive data indicating the location of movable object 10. Controller 22may communicate with various other types of devices, including abarometer, an inertial measurement unit (IMU), a transponder, or thelike, to obtain positioning information and velocity information ofmovable object 10. Controller 22 may also provide control signals (e.g.,in the form of pulsing or pulse width modulation signals) to one or moreelectronic speed controllers (ESCs) (not pictured), which may beconfigured to control one or more of propulsion devices 12. Controller22 may thus control the movement of movable object 10 by controlling oneor more electronic speed controllers.

Terminal 32 may be configured to receive input, such as input from auser (i.e., user input), and communicate signals indicative of the inputto controller 22. Terminal 32 may be configured to receive input andgenerate corresponding signals indicative of one or more types ofinformation, such as control data (e.g., signals) for moving ormanipulating movable device 10 (e.g., via propulsion devices 12),payload 14, and/or carrier 16. Terminal 32 may also be configured toreceive data and information from movable object 10, such as datacollected by or associated with payload 14 and operational data relatingto, for example, positional data, velocity data, acceleration data,sensory data, and other data and information relating to movable object10, its components, and/or its surrounding environment. Terminal 32 maybe a remote control with physical sticks, levers, switches, and/orbuttons configured to control flight parameters, or may be or include atouch screen device, such as a smartphone or a tablet, with virtualcontrols for the same purposes, and may employ an application on asmartphone or a tablet, or a combination thereof.

In some embodiments, terminal 32 may include a smart eyeglass. As usedherein, a smart eyeglass may include any wearable computer glasses orother wearable item that can provide additional information to an imageor scene that a wearer sees. A smart eyeglass may include an opticalhead-mounted display (OHMD) or embedded wireless glasses withtransparent heads-up display (HUD) or augmented reality (AR) overlaythat has the capability of reflecting projected digital images as wellas allowing the user to see through it or see better with it. The smarteyeglass may serve as a front-end display for images, videos, and otherdata or information received from the movable object 10, for example,via cellular technology or Wi-Fi. In some embodiments, the smarteyeglass may also control the movable object 10 via natural languagevoice commands and/or use of touch buttons on the smart eyeglass.

In the example shown in FIGS. 2A and 2B, terminal 32 may includecommunication devices 34 that facilitate communication of informationbetween terminal 32 and other entities, such as movable object 10 oranother terminal 32. Communication devices 34 may include antennas orother devices configured to send and/or receive signals. Terminal 32 mayalso include one or more input devices 36 configured to receive inputfrom a user for communication to movable object 10. FIG. 2A shows oneexemplary embodiment of terminal 32 having an input device 36 with aplurality of input devices, such as levers 38 and 40, buttons 42, andtriggers 44 for receiving one or more inputs from the user. Each inputdevice of terminal 32 may be configured to generate an input signalcommunicable to controller 22 and usable by controller 22 as inputs forprocessing. In addition to flight control inputs, terminal 32 may beused to receive user inputs of other information, such as manual controlsettings, automated control settings, control assistance settings etc.,which may be received, for example, via buttons 42 and/or triggers 44.It is understood that terminal 32 may include other or additional inputdevices, such as buttons, switches, dials, levers, triggers, touch pads,touch screens, soft keys, a mouse, a keyboard, a voice recognitiondevice, and/or other types of input devices. It is understood thatdifferent combinations or layouts of input devices for a terminal suchas terminal 32 are possible and within the scope of this disclosure.

As shown in the alternative embodiment of FIG. 2B, terminal 32 may alsoinclude a display device 46 configured to display information to and/orreceive information from a user. For example, terminal 32 may beconfigured to receive signals from movable object 10, which signals maybe indicative of information or data relating to movements of movableobject 10 and/or data (e.g., imaging data) captured by movable object 10(e.g., in conjunction with payload 14). In some embodiments, displaydevice 46 may be a multifunctional display device configured to displayinformation on a multifunctional screen 48 as well as receive user inputvia the multifunctional screen 48. For example, in one embodiment,display device 46 may be configured to receive one or more user inputsvia multifunctional screen 48. In another embodiment, multifunctionalscreen 48 may constitute a sole input device for receiving user input.

In some embodiments, terminal 32 may be or include an interactivegraphical interface for receiving one or more user inputs. That is,terminal 32 may provide a graphical user interface (GUI) and/or includeone or more graphical versions of input devices 36 for receiving userinput. Graphical versions of terminal 32 and/or input devices 36 may bedisplayable on a display device (e.g., display device 46) or amultifunctional screen (e.g., multifunctional screen 48) and may includegraphical features, such as interactive graphical features (e.g.,graphical buttons, text boxes, dropdown menus, interactive images,etc.). For example, in some embodiments, terminal 32 may includegraphical representations of input levers 38 and 40, buttons 42, andtriggers 44, which may be displayed on and configured to receive userinput via multifunctional screen 48. In some embodiments, terminal 32may be configured to receive all user inputs via graphical inputdevices, such as graphical versions of input devices 36. Terminal 32 maybe configured to generate graphical versions of input devices 36 inconjunction with a computer application (e.g., an “app”) to provide aninteractive interface on the display device or multifunctional screen ofany suitable electronic device (e.g., a cellular phone, a tablet, etc.)for receiving user inputs.

In some embodiments, the display device (e.g., 46) may display an imagereceived from movable object 10 and include interactive means for theuser to identify or select a portion of the image of interest to theuser. For example, display device 46 may include a touchscreen so thatthe user can identify or select the portion of interest by touching thecorresponding part of the touchscreen.

In some embodiments, display device 46 may be an integral component ofterminal 32. That is, display device 46 may be attached or fixed toterminal 32. In other embodiments, display device may be connectable to(and dis-connectable from) terminal 32. That is, terminal 32 may beconfigured to be electronically connectable to display device 46 (e.g.,via a connection port or a wireless communication link) and/or otherwiseconnectable to terminal 32 via a mounting device 50, such as by aclamping, clipping, clasping, hooking, adhering, or other type ofmounting device.

In some embodiments, terminal 32 may be configured to communicate withelectronic devices configurable for controlling movement and/or otheroperational aspects of movable object 10. For example, display device 46may be a display component of an electronic device, such as a cellularphone, a tablet, a personal digital assistant, a laptop computer, orother device. In this way, users may be able to incorporate thefunctionality of other electronic devices into aspects of controllingmovable object 10, which may allow for more flexible and adaptablecontrol schemes. For example, terminal 32 may be configured tocommunicate with electronic devices having a memory and at least oneprocessor and can be used to provide user input via input devicesassociated with the electronic device (e.g., a multifunctional display,buttons, stored apps, web-based applications, etc.). Communicationbetween terminal 32 and electronic devices may also be configured toallow for software update packages and/or other information to bereceived and then communicated to controller 22 (e.g., via communicationsystem 20), shown in FIG. 1.

Although not shown in the figures, the remote control may comprise otherforms of control devices, such as a helmet, a goggle, or other devices,that allow user input and communicate the user input to movable object10 for controlling the movements thereof, as well as display views ofvision subsystem 104 (for example, images captured by the on-boardcamera).

FIG. 3 shows an exemplary controller 22, implemented as system-on-chip(SoC) controller 300, that may include a flight control subsystem 102coupled to communicate with a vision subsystem 104. Vision subsystem 104may be configured to detect and visualize (e.g., using computer vision)objects surrounding the UAV. Flight control subsystem 102 may receiveinformation from vision subsystem 104 and utilize the information todetermine a flight path or make adjustments to an existing flight path.For example, based on the information received from vision subsystem104, flight control subsystem 102 may decide whether to stay on anexisting flight path, change the flight path to track an objectrecognized by vision subsystem 104, or change the flight path (e.g.,override a command received from an operator) to avoid an obstacledetected by vision subsystem 104.

It is contemplated that vision subsystem 104 may utilize various typesof instruments and/or techniques to detect objects surrounding the UAV.For instance, in some embodiments, vision subsystem 104 may communicatewith an ultrasonic sensor 120 configured to detect objects surroundingthe UAV and measure the distances between the UAV and the detectedobjects. Vision subsystem 104 may communicate with other types ofsensors as well, including time of flight (TOF) sensors 122, radars(e.g., including millimeter wave radars), sonars, lidars, barometers, orthe like.

Vision subsystem 104 may be coupled to communicate with an imagingsubsystem 106. Imaging subsystem 106 may be configured to obtain imagesand/or video footage using one or more imaging devices (e.g., cameras)124. Vision subsystem 104 may utilize the images or video footage togenerate a visual representation of the environment surrounding the UAV.It is contemplated that such a visual representation may be utilized forvarious purposes. In some embodiments, for example, vision subsystem 104may process the visual representation using one or more imagerecognition or computer vision processes to detect recognizable objects.Vision subsystem 104 may report objects recognized in this manner toflight control subsystem 102 so that flight control subsystem 102 candetermine whether or not to adjust the flight path of the UAV. Inanother example, vision subsystem 104 may provide (e.g., transmit) thevisual representation to a remote operator so that the remote operatormay be able to visualize the environment surrounding the UAV as if theoperator was situated onboard the UAV. In still another example, thevisual representation may be recorded in a data storage device locatedonboard the UAV.

In some embodiments, flight control subsystem 102, vision subsystem 104,imaging subsystem 106, and imaging device 124 may be configured tooperate with reference to a common time signal. In some embodiments,flight control subsystem 102 may be configured to provide asynchronization (SYNC) signal to one or more of vision subsystem 104,imaging subsystem 106, and imaging device 124. Flight control subsystem102 may use the SYNC signal to control the timing of exposures (orrecordings) of imaging device 124, may determine metadata (e.g.,location, altitude, heading, temperature, etc.) at the time the SYNCsignal was sent, and may timestamp the metadata accordingly. Visionsubsystem 104 may then associate the metadata with the captured imagesor video footage based on the timestamp.

In some embodiments, the images or video footage captured by imagingdevice 124 may be in a data format which may require further processing.For example, data obtained from an image sensor may need to be convertedto a displayable format before a visual representation thereof may begenerated. In some embodiments, imaging subsystem 106 may process thecaptured footage into the right format. Alternatively or additionally,imaging device 124 or vision subsystem 104 may include one or moreprocessors configured to process the captured images or video footageinto a suitable format for generation of visual representation.

Vision subsystem 104 may utilize the images or video footage to detectobjects surrounding the UAV and report information regarding thedetected objects to flight control subsystem 102. Vision subsystem 104may timestamp the report using the same timestamp originally used forthe captured footage. In this manner, flight control subsystem 102 maybe able to determine what the environment surrounding the movable object10 looked like at a given time and adjust the flight path accordingly ifneeded. Flight control subsystem 102 may also cross-reference locationdata received from other devices (e.g., positioning system 112) againstimage data received from vision subsystem 104 based on timestamps forbetter adjustments of the flight path.

Controller 300 may further include a gimbal control subsystem 108 thatcontrols a gimbal (e.g., carrier 16). Gimbal control subsystem 108 maybe in communication with other subsystems (e.g., flight controlsubsystem 102 and/or imaging subsystem 106). If, for example, imagingsubsystem 106 needs to acquire a 360° panoramic view of the environmentsurrounding movable object 10, gimbal control subsystem 108 may controlthe gimbal to rotate about a vertical axis at a particular rotationalspeed. In another example, if flight control subsystem 102 receives acommand (e.g., from a user or operator) to acquire images or videofootage of a particular location, flight control subsystem 102 mayinstruct gimbal control subsystem 108 to rotate the gimbal so that animaging device (e.g., sensory device 19) mounted on the gimbal pointstoward that particular location. In some embodiments, flight controlsubsystem 102 may communicate with a positioning system (e.g., GPS,GLONASS, Galileo, Beidou, GAGAN, RTK, etc.) to locate the particularlocation and may use the location data to control rotation of thegimbal.

In some embodiments, flight control subsystem 102, vision subsystem 104,imaging subsystem 106, and gimbal control subsystem 108 may be packagedtogether to form blocks (or cores) of single system-on-chip controller300. Alternatively, these subsystems may be packaged and/or grouped inmultiple chips.

Consistent with embodiments of the present disclosure, a movable objectmay have a main camera with an adjustable focus and an auxiliaryfocusing module that facilitates the focusing of the main camera. Theauxiliary focusing module may adopt one of the faster focusingtechniques, such as distance measurement or phase detection, todetermine the proper focal length for the main camera. Once the properfocal length is determined, the movable object's controller may controlthe main camera to adjust its focus accordingly. An auxiliary focusingmodule separate from the main camera may permit use with existing maincameras without modification, increase speed of autofocusing, andprovide great flexibility, as discussed below.

Referring to the figures, sensory device 19 (FIG. 1) may include a maincamera (such as imaging device 124 in FIG. 3) with an adjustable focusconfigured to capture one or more of images and videos. Sensing system18 (FIG. 1), in the meantime, may include an auxiliary focusing modulethat facilitates the autofocusing of the camera in sensory device 19.FIG. 4 illustrates one such example, in which sensory device 19 may beprovided as a main camera 52 with an adjustable focus, and sensingsystem 18 may include an auxiliary focusing device 54. Auxiliaryfocusing device 54 may include, for example, a vision sensor 56 (such asa camera) and a distance measurement unit 58. Distance measurement unit58 may include a directional source that emits an infrared laser pulse,or any other laser pulse or beam at a desired frequency, towards anobject, and receive light beams reflected off the object, and determinedistance based on time-of-flight.

The auxiliary focusing device 54 may be embedded in or attached tocamera 52. Alternatively, the auxiliary focusing device 54 may be astand-alone device that coordinates with the imaging system formeasuring a distance. In the example shown, main camera 52 and auxiliaryfocusing device 54 may be separately mounted on carrier 16 and movableobject 10, respectively. In alternative embodiments, main camera 52 andauxiliary focusing device 54 may be mounted on the same carrier orstructure, through supporting structures such as gimbals. The mountingof main camera 52 and auxiliary focusing device 54 may provide forrelative change in position or orientation with respect to each other.Alternatively, main camera 52 and auxiliary focusing device 54 may bemounted on separate gimbals, which may or may not provide the samedegree of freedom for their respective movements. Further vision sensor56 and distance measurement unit 58 may be provided on the same mountingstructure (such as a gimbal) or on separate mounting structrues.

A remote control consistent with embodiments of the present disclosuremay consist of, for example, remote control 32 described above inconnection with FIGS. 2A and 2B. When a display is included in theremote control or a separate device (such as a computer or smartphone),such display may display images taken by the main camera or the visionsensor, transmitted wirelessly from the movable object to the remotecontrol. When vision sensor 56 comprises a camera, either the maincamera or the vision sensor may provide the first-person view (FPV) onthe display for user's control of the movable object.

Consistent with embodiments of the present invention, auxiliary focusingdevice 54 may assist in determination of the focus of main camera 52.For example, distance measurement unit 58 may measure distance to anobject, and main camera 52 may adjust its focus according to themeasured distance. In some embodiments, vision sensor 56 may capture aview or image of the surrounding of movable object 10. The visionsubsystem 104 of movable object 10 may detect an object within the viewor image. Distance measurement unit 58 can then measure the distance tothe detected object. Alternatively, the captured view or image may betransmitted to terminal 32 for display to the user or operator. The usermay identify an object of interest through controls on terminal 32,which may transmit the user's identification to movable object 10.Distance measurement unit 58 can then measure the distance to the objectidentified by the user. In one aspect, execution of the determination offocus and the adjustment of focus by separate components (e.g. by theauxiliary focusing device and by main camera 52, respectively) may allowmain camera 52 to adjust its focus to fall on objects outside the ofcamera's view, thereby providing greater flexibility than traditionalautofocusing methods. Vision sensor 56 may further track the detected oridentified object, such that distance measurement unit 58 continues tomeasure the distance to the object and the focus of main camera 52remains on the object when the object moves in or out of the view ofmain camera 52. Depending on the applications, the detected oridentified object may be anything of interest, for example a nonmovingobject, such as a tree, or a moving object, such as a vehicle or aperson or even a part of a person's face.

Reference is now made to FIG. 5, which illustrates steps of anautofocusing process 500, consistent with embodiments of the presentdisclosure. For purposes of explanation and not limitation, process 500may be performed by software executing in controller 300 and/or movableobject 10.

In step 502, an area of interest may be identified. This area ofinterest may be an object, such as a tree, a landmark, a person, or theface of a person, in the view or image captured by the vision sensor inthe auxiliary autofocusing device. Identification of the area ofinterest may be achieved through object detection or user designation ona display of the view or image on the remote terminal. The area ofinterest may also be identified through predetermination, for example,set by controller 300.

In step 503, the area of interest may be tracked. Tracking the area ofinterest may be achieved automatically by image processing to identifythe movement of the area of interest in the view of vision sensor 56, orby the user viewing such movement and exercising corresponding controlson terminal 32.

In step 504, based on tracking of the measurement, auxiliary focusingdevice 54 may be adjusted to prepare for distance measurement. Forexample, if distance measurement unit 58 includes a directional source,such as a laser or ultrasound generator, the directional source may befirst tuned or adjusted to face the area of interest, based on theposition of the area of interest in the view of vision sensor 56. Ifdistance measurement unit 58 is movable within auxiliary focusing device54, then the adjustment of the directional source may be achievedthrough controlled movement of distance measurement unit 58. If distancemeasurement unit 58 is not movable but auxiliary focusing device 54 ismovable within movable object 10, then the adjustment of the directionalsource may be achieved through controlled movement of auxiliary focusingdevice 54. If neither movement of distance measurement unit 58 withinauxiliary focusing device 54 nor movement of auxiliary focusing device54 within movable object 10 is permitted, the adjustment of thedirectional source may be achieved through controlled movement ofmovable object 10 by, for example, controlling propulsion devices 12 ofmovable object 10 to adjust the spatial disposition, velocity, and/oracceleration of the movable object 10 with respect to six degrees offreedom (e.g., three translational directions along its coordinate axesand three rotational directions about its coordinate axes) to enablemovable object 10 to automatically track the target. If more than one ofthese components can move with respect to one another, then acombination of controlled movements thereof may be used to achieve thedesired adjustment or tuning of the directional source.

Once the directional source is adjusted to face the area of interest,distance may be measured based on, for example, time of flight in step505. In particular, the distance to the area of interest from theauxiliary focusing device 54/main camera 52 may be calculated based on atotal time of the light beam traveling back and forth between the areaof interest and the auxiliary focusing device, and the speed of theemitted wave such as light, infrared signal, or ultrasound.

Steps 504 and 505 may be repeated as needed. For example, after distancemeasurement in step 505, step 504 may be performed again to fine adjustauxiliary focusing device 54 to better orient it towards the area ofinterest, after which step 505 may be performed again to achieve bettermeasurement accuracy, and so on.

In step 506, the distance and position of the area of interest relativeto main camera 52 may be determined. In particular, as discussed indetail below, the position of the area of interest relative to auxiliaryfocusing device 54 and the position or posture of auxiliary focusingdevice 54 relative to main camera 52 may be used to make suchdetermination. Such determination may be performed in controller 22 ormay be distributed across multiple processors located within auxiliaryfocusing device, main camera 52, and/or other places of movable object10.

In step 507, the focus of camera 52 may be adjusted based on thedetermined relative distance and position of the area of interest. Inone aspect, camera 52 may include a mechanism for automaticallyadjusting its own focus. For example, camera 52 may include softwarecontrol that adjusts the positions of the lenses based on the distanceand position of the area of interest. In other aspects, controller 22may control camera 52 to adjust the focus.

As an example, FIGS. 6A and 6B illustrate the determination of therelative distance and position between the area of interest, e.g., atarget object P_(T), and auxiliary focusing device 54. In the exampleshown, vision sensor 56 may include a camera positioned at point D₁, anddistance measurement unit 58 may include a laser positioned at point D₂.Distance measurement unit 58 may project a laser point at P_(L) on ornear the target object P_(T) and may measure the time of flight to andfrom P_(L). FIG. 6B shows an exemplary view of the camera of visionsensor 56. Laser point P_(L) corresponds to point O_(L) in the camera'sview, as detected by the sensors in the camera. The center of cameraview (or the principal point) O_(C) corresponds to a point P_(C) in theplane of the object. The target object P_(T) as identified by the usercorresponds to point O_(T) in the view.

As shown in FIGS. 6A and 6B, the camera in vision sensor 56 and distancemeasurement unit 58 may be displaced from each other and need notnecessarily be parallel to each other; and the laser point P_(L) doesnot coincide with the center of the view P_(C) or the position of targetobject P_(T). Thus, measurement of distance z may not accurately reflectthe distance to target object P_(T), in which case auxiliary focusingdevice 54 may need to adjust itself or distance measurement unit 58 tofocus the laser on the target object to obtain more accuratemeasurement. Such adjustment needs to be based on the positionalrelationship of the camera and distance measurement unit 58.

In one aspect, the adjustment to focus the laser on the target objectmay be achieved through comparison of the relative positions of thelaser point P_(L), the center of view P_(C) of the vision sensor, thetarget object P_(T), and/or their corresponding positions, i.e., pixelcoordinates, on the image captured by the camera's sensor. The pixelcoordinates (u₀, v₀) of the principal point O_(C) (corresponding toP_(C)) are known. The pixel coordinates (u_(T), v_(T)) of the targetobject P_(T) as projected on the sensor are also known, because the useridentifies the target object on the sensor image.

The positional relationship between the laser and the vision sensor canbe used to find the pixel coordinates (u_(L), v_(L)) of the laser pointP_(L). As the laser point only measures a dimensionless distance z,P_(L) may be initially represented in the laser's own framework asthree-dimensional coordinates P_(L) ^(L)=(0,0, z). For ease ofdescription, superscripts, such as the letter z in the top right cornerof P_(L) ^(L), indicate the coordinate system or frame of reference inwhich the parameter is defined. When the superscript has two letters,the parameter describes the relationship between the two coordinatesystems or frames of reference indicated by the individual letters. Inthe vision sensor's perspective, i.e., in a coordinate system definedwith the vision sensor's position and orientation, the three-dimensionalcoordinates P_(L) ^(V) of the laser point P_(L) may be represented as atransformation of P_(L) ^(L):

P _(L) ^(V) =R ^(VL) P _(L) ^(L) +T ^(VL)   (1)

wherein R^(VL) is a matrix describing the rotational relationship fromthe laser's framework to the vision sensor's framework at distance z;and T^(VL) is a matrix describing the translational relationship betweenthe two frameworks, also at distance z.

R^(VL) and T^(VL) may be determined based on the relative positions ofthe laser and the camera of auxiliary focusing device 54, which areknown to movable object 10. Because R^(VL) and T^(VL) describe thetransformation between the laser and the camera at distance z, the twomatrices necessarily depend on and vary with distance z.

When a camera (in the vision sensor) captures an image, a real point inspace is projected as an imaginary point from the view point of thecamera's sensor. The coordinates of the real point and the coordinatesof the imagery point are related through an intrinsic matrix K of thecamera:

$\begin{matrix}{K = \begin{bmatrix}\alpha_{x} & \gamma & u_{0} \\0 & \alpha_{y} & v_{0} \\0 & 0 & 1\end{bmatrix}} & (2)\end{matrix}$

where a_(x)=fm_(x), a_(y)=fm_(y), f is camera focal length, m_(x) andm_(y) are the scaling factors along the x and y axes, y is the skewparameter between x and y axes, and (u₀, v₀) are coordinates of theprincipal point O_(c). Thus, the location P_(L)* of the imaginary point,as viewed from the camera's sensor through its lenses, can be expressedas:

$\begin{matrix}{P_{L}^{*} = {\begin{bmatrix}x \\y \\z\end{bmatrix} = {KP_{L}^{V}}}} & (3)\end{matrix}$

On the image captured by the sensor inside the camera of the visionsensor, the corresponding point is

$\begin{matrix}{O_{L} = {\begin{bmatrix}u_{L} \\v_{L} \\1\end{bmatrix} = \begin{bmatrix}{x/z} \\{y/z} \\1\end{bmatrix}}} & (4)\end{matrix}$

Once the location of the laser point on the image is determined, thedifference Δ between the laser point and the location of the targetobject can be determined:

$\begin{matrix}{\Delta = {\begin{bmatrix}{\Delta u} \\{\Delta v}\end{bmatrix} = \begin{bmatrix}{u_{T} - u_{L}} \\{v_{T} - v_{L}}\end{bmatrix}}} & (5)\end{matrix}$

If, through tracking of the object or area of interest, the camera ofauxiliary focusing device 54 adjusts its view so that the target objectis at the center of the camera's view, then O_(T) coincides with O_(C),and

$\begin{matrix}{\Delta = {\begin{bmatrix}{\Delta u} \\{\Delta v}\end{bmatrix} = \begin{bmatrix}{u_{0} - u_{L}} \\{v_{0} - v_{L}}\end{bmatrix}}} & (6)\end{matrix}$

In some aspects, having the camera track the target object and adjustitself to keep the target object at its center may simplify theadjustment of the laser to be oriented towards, or focus on, the targetobject, especially where the laser initially projects in an area at adistance away from the target object.

The difference Δ between the laser point and the location of the targetobject can then be used to adjust the laser to focus on the objectselected by the user, through rotation in both x and y directions:

$\begin{matrix}{\begin{bmatrix}\varphi_{x} \\\varphi_{y}\end{bmatrix} = \begin{bmatrix}{\frac{\Delta u}{w} \cdot {FoV}_{x}} \\{\frac{\Delta v}{h} \cdot {FoV}_{y}}\end{bmatrix}} & (7)\end{matrix}$

where φ_(x) and φ_(y) are the angles of rotation in the x and ydirections, w and h are the width and height of the image size sensed bythe camera sensor, and FoV_(x) and FoV_(y) are the fields of view in thex and y directions. For example, if the diagonal field of view of thecamera sensor is FoV, then

$\begin{matrix}{{FoV_{x}} = {2\mspace{11mu}{\tan^{- 1}\left( {{\tan\left( \frac{FoV}{2} \right)} \cdot \frac{w}{\sqrt{w^{2} + h^{2}}}} \right)}}} & \left( \text{8-1} \right) \\{{{Fo}V_{y}} = {2\mspace{11mu}{\tan^{- 1}\left( {{\tan\left( \frac{FoV}{2} \right)} \cdot \frac{w}{\sqrt{w^{2} + h^{2}}}} \right)}}} & \left( \text{8-2} \right)\end{matrix}$

As mentioned above, the transformation matrices, R^(VL) and T^(VL)depend on the distance z and the adjustment to focus the laser point onthe area of interest may affect the measurement of distance z.Therefore, distance measurement unit 58 may perform another measurementafter an adjustment to obtain a more accurate value of distance z. Afterthe updated measurement, a further, finer adjustment may be made basedon the methods described. This measurement-adjustment cycle may berepeated several times to achieve satisfactory accuracy.

Because moveable object 10 and its various components may be in motioncollectively and/or with respect to one another, and the target objectmay be moving too, the distance measurement may be affected bydisturbances or noise caused by such movements. Consistent withembodiments of the present disclosure, multiple distance measurementsmay be repeated over time and combined to obtain more precisedetermination of the distance. Examples of such processing (or datafusion) may be, for example, averaging, filtering, or else. Commonlyknown filters such as Kalman filter, Butterworth filter, or else, may beused.

As an example, a Kalman filter may be applied to the distancemeasurements to reduce the impact of noise. As is known in the art,Kalman filter is a recursive filter that provides a prediction of thestate of variables based on observations. The state of a movable object,for example, can be defined with two observables, the object's positionx (a three-dimensional vector) and velocity {dot over (x)} (derivativeof x over time). A state space X can be defined with the two observablescombined:

$\begin{matrix}{X = \begin{bmatrix}x \\\overset{.}{x}\end{bmatrix}} & (9)\end{matrix}$

The state space X_(k) at time k can be predicted from the state spaceX_(k−1) at time k−1 based on Newton's laws of motion:

X _(k) =FX _(k−1) +Ga _(k)   (10)

where

$F = {{\begin{bmatrix}1 & {\Delta t} \\0 & 1\end{bmatrix}\mspace{14mu}{and}\mspace{14mu} G} = {\begin{bmatrix}{\Delta\;{t^{2}/2}} \\{\Delta\; t}\end{bmatrix}.}}$

Here, a_(k) is the object's acceleration between time k−1 and time kthat may have been caused by uncontrolled forces or disturbances, suchas wind, slipping on slippery road, etc. As noise to the otherwiseprecise prediction based on Newton's laws of motion, a_(k) is assumed tohave a normal (Gaussian) distribution with a mean value of 0 and astandard deviation of σ_(a), i.e., a_(k)˜N (0, σ_(a)).

In addition to the noise during the prediction, when the state space Xis observed or measured, more noise may be introduced, such that themeasurement Z at time k is:

Z _(k) =HX _(k)+ε_(k)   (11)

where H is the measurement matrix that represents the relationshipbetween the state space X and the variables or observables undermeasurement. ε_(k) represents measurement noise, which is also assumedto have a normal distribution with a mean value of 0 and a standarddeviation of σ_(k), i.e., ε_(k)˜N(0, σ_(k)).

Based on measurement Z_(k), a best estimate of X_(k) may be made. Thebest estimate of X_(k) is considered the value of the state space attime k and may also be used in conjunction with formula (10) to predictthe state space at the next time k+1. The Kalman filter uses acovariance matrix P to reflect the accuracy of the prediction and bestestimate of the state space. The covariance matrix at time k−1 is usedin combination with other information, such as measurement Z_(k)(calculated by formula (11)) and the prediction of X_(k) (calculated byformula (10)), to find the best estimate of X_(k). Like the state space,a prediction of the covariance matrix at time k, P_(k|k−1), is madebased on information available at time k−1, i.e., measurement Z_(k−1),best estimate of X_(k−1), the covariance matrix at time k−1, and theassumed noise levels. After measurement at time k, the covariance matrixis updated to P_(k|k), which may then be used in determination of thebest estimate of X_(k) and prediction of the covariance matrix for nexttime k+1, P_(k+1|k).

Thus, with more observations over time, the state space and covariancematrix are constantly updated, self-correcting possible accumulation oferrors. Noises are evened out rather than accumulated. The prediction ofthe state space, as well as the best estimate of the current state spacebased on the prediction and measurement, become more accurate.

Consistent with embodiments of the present disclosure, a Kalman filtermay be used to estimate the location of the target object with relativeprecision. As in the example discussed above, equations (9) and (10) canbe applied, as the motions of the movable object and the target objectfollow Newton's laws of motion. To find out the distance of the targetobject from the main camera, two variables may be measured or observed,such as the pixel coordinates (U_(T), V_(T)) of the target object on theimage of the sensor, as well as the location P_(T) ^(C) of the targetobject with in the frame of reference of the camera of the auxiliaryfocusing device:

$\begin{matrix}\begin{matrix}{\begin{bmatrix}u_{T} \\v_{T} \\1\end{bmatrix} = {{K\left\lbrack {R^{VW}\left( {P_{T}^{W} - P_{V}^{W}} \right)} \right\rbrack} + \delta_{k}}} \\{= {{K\left\lbrack {R^{VI}{R^{IW}\left( {P_{T}^{W} - \left( {P_{I}^{W} + {R^{WI}T_{V}^{I}}} \right)} \right)}} \right\rbrack} + \delta_{k}}}\end{matrix} & \left( \text{12-1} \right) \\\begin{matrix}{P_{T}^{V} = {{R^{VW}\left( {P_{T}^{W} - P_{V}^{W}} \right)} + \gamma_{k}}} \\{= {{R^{VI}{R^{IW}\left( {P_{T}^{W} - \left( {P_{I}^{W} + {R^{WI}T_{V}^{I}}} \right)} \right)}} + \gamma_{k}}}\end{matrix} & \left( \text{12-2} \right)\end{matrix}$

In formulas (12-1) and (12-2):

-   -   R^(VW) is the matrix describing the rotational relationship from        the world coordinate system to the vision sensor's frame of        reference;    -   P_(T) ^(W) is the location of the target object in the world        coordinate system;    -   P_(V) ^(W) is the location of the vision sensor in the world        coordinate system;    -   δ_(k) is the noise, at time k, in the observation of the pixel        coordinates;    -   R^(VI) is the matrix describing the rotational relationship from        the inertial measurement unit (IMU)'s frame of reference to the        vision sensor's frame of reference;    -   R^(IW) is the matrix describing the rotational relationship from        the world coordinate system to the IMU's frame of reference;    -   P_(I) ^(W) is the location of the IMU in the world coordinate        system;    -   R^(WI) is the matrix describing the rotational relationship from        the IMU's frame of reference to the world coordinate system;

T_(V) ^(I) is the location of the vision sensor in the IMU's frame ofreference (T_(V) ^(I) may be the same as T^(VI), the translationalmatrix between the IMU's frame of reference and the vision sensor'sframe of reference);

-   -   P_(T) ^(V) is the location of the target object in the vision        sensor's frame of reference; and    -   γ_(k) is the noise, at time k, in the observation of the        three-dimensional location of the target object in the vision        sensor's frame of reference.

The noise in the observations or measurements follow a normal (orGaussian) distribution, i.e., δ_(k)˜N(0, σ) and γ_(k)˜N(0, σ^(k)), whereσ and σ_(k) are the standard deviations in the observations of the pixelcoordinates and 3-D location of the target object, respectively, and σ²and σ_(k) ² represent the corresponding variances.

Initial states of the Kalman filter can be configured through severalinitial measurements, for example,

${X_{0} = \begin{bmatrix}x_{0} \\0\end{bmatrix}},$

where x₀ is the average of several measurements of the position of theobject at time 0. The covariance matrix may be initialized to be

$\begin{matrix}{P_{0|0} = \begin{bmatrix}B & 0 \\0 & B\end{bmatrix}} & (13)\end{matrix}$

The value of B may be chosen based on application needs. A greater valueof B gives the earlier measurements more weight, whereas a smaller valueof B weighs later measurements more.

With filtering, the location of the target object P_(T) ^(V) in thecamera's frame of reference can be reliably determined. The location ofthe target object P_(T) ^(M) in the main camera's frame of reference canthen be determined:

P _(T) ^(M) =R ^(MV) P _(T) ^(V) T ^(MV)   (14)

wherein R^(MV) is the matrix representing the rotational relationshipfrom the vision sensor to the main camera 52, and T^(MV) is the matrixdescribing the translational relationship from the vision sensor to themain camera 52.

Once the location of the target object in the main camera's frame ofreference is determined, the desired depth d of focus can be determined.Consistent with embodiments of the present disclosure, the main cameracan focus on the target object. Alternatively, the main camera can focusat the desired depth d regardless of whether the target object is inview of not. For example, the depth d can be determined from thethree-dimensional location P_(T) ^(M) of the target object as:

d=|P _(T) ^(M)|  (15-1)

or

d=(P_(T) ^(M))⁽³⁾   (15-2)

where |P_(T) ^(M)| is the length of the vector represented by P_(T)^(M). (P_(T) ^(M))⁽³⁾ indicates the third component of P_(T) ^(M), i.e.,the z component.

FIG. 7 illustrates another exemplary autofocusing process 700,consistent with embodiments of the present disclosure. For purposes ofexplanation and not limitation, process 700 may be performed by softwareexecuting in controller 300 and/or movable object 10.

In step 701, an area of interest may be identified. This area ofinterest may be an object, such as a tree, a landmark, a person, or theface of a person, in the view or image captured by the vision sensor inthe auxiliary autofocusing device. Identification of the area ofinterest may be achieved through object detection or user designation ona display of the view or image on the remote terminal, at step 701-1.Once the user selects the area of interest, controller 300 or movableobject 10 recognizes the identified target area at step 701-2 andactivates tracking algorithm to keep track of the target at 701-3.

The same target or area of interest will remain selected or tracked, atstep 702, through tracking algorithm.

In step 703, the auxiliary focusing device is adjusted to prepare fordistance measurement. For example, if distance measurement unit 58includes a directional source, such as a laser or ultrasound generator,the directional source may be first tuned or adjusted to face the areaof interest, at step 703-1, based on the position of the area ofinterest in the sensor image or in the view of vision sensor 56. Thisadjustment takes into calculation or account the sensor parameters ofthe auxiliary focusing device and/or posture of the main camera.

The adjustment of the auxiliary focusing device may be fine-tuned, atstep 703-2, after a distance measurement by the distance measurementunit. And the fine tuning may be repeated until data converges, afterwhich data fusion may be performed, at step 704, to reduce the impact ofdisturbances or noise.

Once the distance from the auxiliary focusing device to the targetobject or area is measured, parameters of the target object or area isconverted into the coordinate system of the main camera, at step 705,based on the positional relationship between the main camera and theauxiliary focusing device. At step 706, the depth of field (DOF) of thetarget object or area is calculated in the coordinate system of the maincamera.

Based on the calculated depth of field, the focal distance may bedetermined and then, at step 707, the focus of the main camera may beadjusted.

It is to be understood that the disclosed embodiments are notnecessarily limited in their application to the details of constructionand the arrangement of the components set forth in the followingdescription and/or illustrated in the drawings and/or the examples. Thedisclosed embodiments are capable of variations, or of being practicedor carried out in various ways.

It will be apparent to those skilled in the art that variousmodifications and variations can be made to the disclosed devices andsystems. Other embodiments will be apparent to those skilled in the artfrom consideration of the specification and practice of the discloseddevices and systems. It is intended that the specification and examplesbe considered as exemplary only, with a true scope being indicated bythe following claims and their equivalents.

1-54. (canceled)
 55. A movable object, comprising: a first camera; anauxiliary focusing device including: a vision sensor comprising a secondcamera; and a distance measurement unit; and a processor configured to:cause the vision sensor to capture a view, receive a selection of anarea of interest in the view captured by the vision sensor, cause thedistance measurement unit to measure a distance to the area of interest,and cause adjustment of the first camera's focus based on the measureddistance.
 56. The movable object of claim 55, wherein the movable objectis an unmanned aerial vehicle.
 57. The movable object of claim 55,wherein the auxiliary focusing device is movable with respect to thefirst camera.
 58. The movable object of claim 55, wherein the distancemeasurement unit is movable with respect to the first camera.
 59. Themovable object of claim 55, wherein the vision sensor, the distancemeasurement unit, and the first camera are all movable with respect toone another.
 60. The movable object of claim 55, wherein the distancemeasurement unit measures the distance based on a flight of time. 61.The movable object of claim 55, wherein the distance measurement unitcomprises a laser.
 62. The movable object of claim 55, wherein thedistance measurement unit comprises an ultrasound wave generator. 63.The movable object of claim 55, wherein the processor receives theselection of the area of interest from a user.
 64. The movable object ofclaim 55, wherein the processor is configured to process the distancemeasurement to reduce an impact of noise.
 65. The movable object ofclaim 55, wherein the processor uses a Kalman filter to process thedistance measurement.
 66. The movable object of claim 55, wherein theprocessor is configured to cause adjustment of the distance measurementunit based on the measured distance to orient the distance measurementunit towards the area of interest.
 67. The movable object of claim 66,wherein the processor is configured to cause the distance measurementunit to repeat the distance measurement after the adjustment of thedistance measurement unit.
 68. The movable object of claim 66, whereinthe vision sensor is configured to track the area of interest.
 69. Themovable object of claim 55, wherein the vision sensor tracks the area ofinterest such that the area of interest remains at the center of theview.
 70. The movable object of claim 55, wherein the processor isconfigured to cause the camera to focus based on the measured distanceregardless of whether the area of interest is in the first camera'sview.
 71. The movable object of claim 55, wherein the processor isconfigured to determine the depth of the area of interest based on themeasured distance, and based on a positional relationship between thefirst camera and the distance measurement unit or a positionalrelationship between the first camera and the auxiliary focusing device.72. The movable object of claim 55, wherein the processor is configuredto cause the first camera to focus on the area of interest.